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1 . INTRODUCTION 


1 .1  Summary 

This  report  covers  the  second  year  of  a three-year  program  of 
research  and  development  directed  toward  evaluation  of  information  needs  in 
decision  making  and  control  of  advanced  aircraft.  Its  purpose  is  to 
establish  new  techniques  for  the  selection  of  essential  information  in 
operational  systems.  These  techniques  center  on  the  adaptive  modeling  of 
individual  decision  behavior  to  derive  the  immediate  value  of  information. 

The  program  will  result  in  aids  for  the  real  time  management  of 
communications.  Specific  objectives  of  the  three  year  program  include  the 
following: 

(1)  Establish  a repertoire  of  techniques  to  model  the  information 
needed  for  operation  of  tactical  airborne  systems. 

(2)  Provide  model-based  computer  programs  capable  of  ascertaining 
the  potential  usefulness  of  alternative  information  sources, 
transmission  systems,  and  information  displays. 

(3)  Develop  and  experimentally  validate  a supervisory  program  for 
management  of  communications  in  simulated  tactical  airborne 
systems. 

(4)  Produce  guidelines  for  field  application  of  the  information 
analysis  and  management  programs  in  a variety  of  complex  systems. 

The  initial  year's  work  established  techniques  for  modeling  the 
information  seeking  decisions  necessary  for  continuous  decision  making  and 
control  of  remotely  piloted  vehicles.  The  work  also  experimentally 


demonstrated  the  usefulness  of  adaptive  techniques  for  prediction  and 
analysis  of  Information  seeking  behavior.  The  work  reported  here  builds 
on  these  findings  by  investigating  the  usefulness  of  the  adaptive  decision 
model  for  automating  the  presentation  of  information  and  for  evaluating 
the  effectiveness  of  information  display  configurations. 

1.2  The  Problem 


Corranand  and  control  of  modern  military  systems  involves  selection 
among  increasing  large  amounts  of  information.  Computerized  systems 
typically  make  available  copious  amounts  of  information  concerning 
remaining  resources,  environmental  state,  potential  computer  aiding,  and 
predicted  circumstances  and  actions.  The  costs  of  conmuni cations  and  the 
limited  processing  capabilities  of  the  human  operator  make  it  necessary  to 
optimize  the  information  selected,  transmitted  and  presented. 

The  central  problem  in  performing  an  analysis  of  information  needs  is 
the  structuring  of  the  decisions.  Choices  must  be  modeled  regarding 
variables  such  as  the  mix  of  information  sensing,  processing,  encoding, 
transmitting,  and  display.  Throughout  this  process,  a balance  must  be 
maintained  between  maximizing  operator  awareness  of  system  operation  and 
minimizing  communications  costs  and  operator  load. 

Some  initial  efforts  have  been  made  toward  analyzing  and  automating 
the  communications  management  functions.  Information  and  control  allocation 
techniques  have  been  proposed  using  criteria  based  on  queing  models  (Rouse, 
1975;  Engstrom  and  Rouse,  1976),  optimal  control  models  (Sheridan,  1976; 
Rouse  and  Gopher,  1977),  and  multi-attribute  decision  models  (Steeb  and 
Freedy,  1976).  This  program  represents  an  effort  to  develop,  integrate  and 
implement  the  more  promising  of  these  techniques. 
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In  brief,  the  command  and  control  task  can  be  represented  as  a multi- 
stage decision  task  of  information  acquisition  and  action  selection.  The 
effectiveness  of  the  action  decisions  are  dependent  on  the  appropriateness 


and  timeliness  of  the  information  acquired.  The  choices  of  what  information 
to  transmit  thus  must  reflect  the  task  circumstances,  the  operator's  and 
automatic  system's  capabilities,  and  the  communication  channel 
characteristics.  These  decisions  can  be  expressed  analytically  using  a 
multi-dimensional  set  of  utilities  tied  to  the  potential  consequences  of 
the  actions.  The  weights  of  the  various  dimensions  reflect  the  importances 
placed  by  the  individual  decision  maker  on  each  factor. 

A program  was  developed  along  these  lines,  containing  both  objective 
probability  estimation  algorithms  and  subjective  behavioral  models.  The 
probability  estimation  algorithms  calculate  the  likelihood  of  occurrence 
of  consequences.  Such  algorithms  depend  on  frequency  determinations  and 
optimal  Bayesian  calculations.  The  subjective  models  use  these 
determinations  as  inputs  and  are  adaptive  multi -attribute  forms  tied  to 
prediction  of  behavior.  The  models  demonstrate  potential  for  a number  of 
types  of  aiding.  Among  the  uses  planned  are  evaluation  and  specification  of 
communication  configurations,  individualized  training  in  information 
acquisition,  and  automated  communications  management. 


Testing  of  the  models  was  accomplished  by  determining  the  aiding  and 
analysis  capabilities  provided  in  a advanced  aircraft  task  simulation.  The 
task  simulation  incorporates  many  of  the  functions  involved  in  multiple 
intercept  operations  and  has  extensive  performance  analysis  capabilities. 


1 .4  Current  Objectives 


The  focus  of  this  phase  of  the  research  and  development  program  is 
to  extend  the  application  of  the  adaptive  decision  model  to  include 
information  eva-uation  and  management  functions  in  an  advanced  aircraft 
context.  This  builds  on  the  findings  of  the  initial  year's  work  in  which 
the  adaptive  program  was  found  to  be  a useful  model  for  analyzing  and 
predicting  information  seeking  behavior.  The  specific  objectives  of  this 
second-year  program  include: 

(1)  Extend  the  review  of  the  engineering  and  psychological 
literature  to  include  airborne  information  selection  and 
presentation. 

(2)  Modify  the  multi-attribute  decision  model  developed  for  RPV 
supervision  to  one  capable  of  dealing  with  information  selection 
in  advanced  aircraft.  Additional  factors  for  time  stress  and 

an  expanded  action  set  are  to  be  included. 

(3)  Develop  a task  simulation  resembling  multiple  intercept 
operations  in  advanced  aircraft.  Install  computer  programs  for 
parameter  estimation,  information  value  assessment,  and 
communications  management. 

(4)  Conduct  an  experimental  study  of  the  effectiveness  of  the  multi- 
attribute model  for  information  system  evaluation  and  for 
automated  information  management.  Direct  operator  control  of 
information  .is  used  as  a baseline  for  comparison. 
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1 .5  Report  Organization 

The  organization  of  this  report  is  as  follows:  Chapter  2 reviews 
selected  background  concepts,  traces  the  development  of  the  adaptive  models, 
and  describes  previous  experimental  studies  performed  under  this  program. 
Chapter  3 discusses  the  application  of  information  value  modeling  to 
advanced  aircraft  decision  making  and  control  tasks.  Chapter  4 describes 
the  experimental  plans  and  procedures  for  testing  the  usefulness  of  the 
adaptive  models.  Chapter  5 summarizes  the  results  of  the  experimental 
study.  Chapter  6 closes  with  a discussion  of  the  research  findings  and 
opportunities  for  application. 
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2.  INFORMATION  VALUE  MODELING 

2.1  Overview 

The  selection,  aquisition  and  processing  of  information  are 
activities  that  are  involved  in  virtually  every  aspect  of  tactical  airborne 
operations.  The  operator  must  continuously  maintain  an  awareness  of  the 
aircraft  state,  the  environment,  the  capacity  and  quality  of  the 
communication  channels,  and  the  progress  toward  objectives.  This  section 
will  explore  the  more  important  techniques  of  modeling  these  information- 
related  activities. 


Information  Seeking  Models 


2.2.1  Cue  Regression.  The  cue  regression  approaches,  exemplified  by 
Brunswik  (1940,  1952),  Dawes  and  Corrigan  (1974),  and  Anderson  and  Shanteau 
(•'970),  are  the  more  descriptive  of  the  two  methodologies.  In  cue 
regression,  a single  linear  model  aggregates  a variety  of  independent 
factors  to  predict  behavior.  Each  factor  is  scaled  and  weighted  according 
to  importance.  The  influences  of  the  individual  factors  are  then  combined 
additively,  multipl icatively,  or  in  some  other  fashion  to  arrive  at  an 
evaluation  of  each  information  choice.  The  formulation  for  the  simple 
additive  case  is: 


R = l W.S.  + C (2-1) 

Where  R is  an  interval  scaled  performance  response,  Si  is  the  stimulus  level 
on  dimension  i,  Wi  is  the  regression  weight,  and  C is  an  optional  scaling 
constant.  The  advantages  of  the  cue  regression  approach  are  its  simplicity 
and  robustness.  Virtually  any  available  predictive  factors  can  be  used, 
such  as  information  content,  data  perishability,  channel  quality,  and 
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operator  load.  In  fact,  the  ensuing  action  decision  does  not  even  have  to 
be  represented.  The  terminal  decision  is  simply  assumed  to  drive  the 
operator's  information  seeking  behavior.  Also,  the  linear  model  has  been 
shown  to  be  robust  to  inaccuracies  in  modeling,  and  highly  predictive  of 
behavior  in  a variety  of  situations  (Dawes  and  Corrigan,  1974;  Slovik, 
Fischhoff,  and  Lichtenstein,  1977). 

2.2.2  Multi-Attribute  Utility  Models.  The  mul ti-attribute  utility  (MAU) 
models,  pioneered  by  Raiffa  and  his  colleagues  (Raiffa,  1969;  Keeney  and 
Raiffa,  1975)  and  by  V.  Winterfeldt  (1975),  tend  to  legitimize  the  process 
begun  by  the  cue  regression  models.  The  MAU  models  make  the  information 
seeking  process  more  goal  directed,  normative  and  axiomatic.  Instead  of 
simply  attempting  to  predict  behavior  on  the  basis  of  a set  of  independent 
features,  the  utility  models  tie  the  information  decisions  directly  to  the 
ensuing  action  decisions.  The  value  of  obtaining  information  is  determined 
by  calculating  its  impact  on  the  expected  utility  of  the  subsequent  action 
decision.  The  information  is  assumed  to  change  the  probability 
distributions  of  the  consequence  sets  and,  in  turn,  to  revise  the  expected 
values  of  the  alternative  actions.  Nevertheless,  the  form  of  the  model  is 
again  a linear  additive  rule.  The  utility  of  an  action  is  considered  to 
be  an  aggregate  of  many  possible  outcomes,  each  expressed  along  a set  of 
attributes: 


EU  (ak)  = l P(zh)  l U1  (ak>zn)  (2-2) 

states  attributes 
h i 

Where  EU(a. ) is  the  expected  utility  of  action  k,  P(z  ) is  the  probability 

k n . h 

of  state  zn  occurring,  and  Uj(ak,z  ) is  the  utility  function  over  the  i u 
attribute  associated  with  state  h and  action  k.  The  formulation  is  the 
result  of  several  key  simplifying  assumptions.  The  decision  maker  is 


assumed  to  be  risk  neutral,  so  that  he  is  indifferent  between  the 


expectation  across  a set  of  uncertain  outcomes  and  the  uncertain  outcomes 
themselves.  This  allows  the  probabilities  to  be  entered  as  simple 
coefficients.  Also,  the  attributes  are  assumed  to  satisfy  additive 
independence,  allowing  the  linear  additive  form  of  aggregation.  Tests  for 
compliance  with  these  assumptions  can  be  found  in  V.  Winterfeldt  (1975) 
or  Keeney  and  Raiffa  (1976). 

I fT 

The  impact  of  a message  or  item  of  data  is  to  change  the  probability 
distribution  of  the  states  z h-  Once  the  message  is  received,  a maximum 
utility  action  a*(y)  can  be  identified.  The  expected  utility  of  selecting 
an  information  source  S then  becomes  (Emery,  1969): 

EU  (S)  = l l P(zh)  P(y|zh)  u(a*(y),zh)  (2-3) 

messages  states 

y z 

II 

Here  u(a*(y),zh)  is  the  utility  of  taking  action  a*(y)  given  that  state 
zh  occurs.  The  utility  function  is  again  multi-attributed,  but  for 
simplicity  u(a*(y),zh)  is  portrayed  as  having  already  been  aggregated  across 
the  various  dimensions. 

This  type  of  analysis,  championed  by  such  researchers  as  Emery  (1969), 
Marshak  (1971)  and  Wendt  (1969),  is  suited  for  highly  structured  tasks.  Not 
only  must  the  possible  states,  messages,  actions,  and  outcomes  be  specifiable, 
but  the  prior  state  probabilities  and  the  conditional  probabilities 
characterizing  the  information  system  must  be  derivable.  The  sequence  of 
decision  stages  can  be  depicted  using  a decision  tree,  as  shown  in  Figure 
2-1.  The  tree  is  folded  back  by  associating  with  each  possible  message  the 
maximum  expected  utility  of  the  subsequent  actions.  This  folding  back 
represents  graphically  the  process  of  EU  maximization.  The  favored 
information  source  S is  then  identified  by  comparing  the  expectations  taken 
over  all  possible  messages. 


Should  these  requirements  be  satisfied,  diverse  aids  are  possible 
with  the  utility  model.  Alternative  configurations  for  communications  can 
be  compared  according  to  their  contribution  to  the  attainment  of  immediate 
system  objectives.  The  operator's  consistency  between  information  seeking 
and  action  decisions  can  be  ascertained  and  corrective  feedback  given. 

Also,  the  automated  management  of  communications  can  be  based  on  the 
expected  impact  on  action  effectiveness,  rather  than  on  the  simple  mimicry 
of  operator  behavior  provided  by  the  cue  regression  approach. 

2.2.3  Other  Methodologies.  A number  of  other  techniques  have  also  been 
proposed  to  model  information  seeking  behavior.  Among  these  are  information 
theory  models,  (Whittemore  and  Yovits,  1973),  optimal  control  formulations, 
(Sheridan,  1976;  Rouse  and  Gopher  1977),  queing  models,  (Rouse,  1975; 

Engstrom  and  Rouse,  1976),  and  information  integration  techniques  (Anderson 
and  Shanteau,  1970).  For  the  most  part,  these  techniques  demand  rigid 
problem  structuring  and  continuous  variables.  More  often,  the  communication 
decision  is  incompletely  defined  and  involves  choices  among  discrete  rather 
than  continuous  alternatives.  Thus  the  discrete  operators  used  in  cue 
regression  and  multi-attribute  utility  models  - matrices,  difference 
operators,  and  detailed  parameter  enumerators  - are  more  appropriate.  The 
interested  reader  is  directed  to  the  previous  technical  report  (Steeb,  Chen 
and  Freedy,  1977)  for  a more  detailed  examination  of  these  approaches. 

2.3  Adaptive  Estimation  Techniques 

2.3.1  Background.  The  initial  year  of  work  on  this  program  resulted  in 
development  of  a methodology  for  adaptive  estimation  of  information  value 
parameters.  The  methodology  is  based  on  the  use  of  multi -attribute  utility 
theory  to  incorporate  the  various  objective  and  subjective  factors  that  enter 
into  the  information  decision.  The  adaptive  nature  of  the  program  derives 
from  the  use  of  a training  algorithm  based  on  pattern  recognition  techniques 
to  derive  certain  of  the  model  parameters. 


This  section  traces  the  structuring  of  an  adaptive  information 
seeking  model  within  a general  multi -attribute  utility  framework: 
definition  of  the  attribute  set,  determination  of  model  form,  specification 
of  attribute  levels  and  estimation  of  importance  weights.  The  specific 
application  of  this  methodology  to  aiding  in  advanced  aircraft  operations 
will  be  described  in  chapter  three. 

2.3.2  Factor  Choice.  It  was  noted  in  the  previous  study  (Steeb,  Chen  and 
Freedy,  1977)  that  the  attribute  set  should  be  accessible,  monotonic, 
independent,  complete  and  meaningful.  Also,  a single  set  must  account  for 
both  information  acquisition  and  action  selection  behavior.  Finally,  the 
attribute  set  must  be  manageably  small  in  dimension.  With  these 
considerations  in  mind,  an  initial  taxonomy  of  consequences  can  be 
organized  around  the  following  five  areas: 

(1)  Communications  costs  - such  as  energy,  equipment,  and  attention. 

(2)  Equipment  attrition  - fuel  expenditures,  vehicle  damage,  etc. 

(3)  Objectives  attainment  - area  reconnoitered,  payload  delivered 

(4)  Dynamic  effects  - effects  on  availability  of  future  information 
and  on  system  capabilities. 

(5)  Subjective  factors  - preferences  regarding  control  continuity, 
operator  load. 

A useful  consequence  set  might  contain  a single  dimension  or 
attribute  from  each  of  these  categories. 

2.3.3  Attribute  Level  Determination.  The  level  or  quantity  of  each 
attribute  for  a given  outcome  can  be  determined  in  several  ways.  For 
example,  mappings  between  predictive  features  and  the  attributes  can  be 
established  by  observation  and  adjustment.  Here,  data  available  to  the 
decision  program  concerning  the  environmental  state,  vehicle  state,  channel 


characteristics,  sensor  capabilities,  and  operator  load  can  be  used  to 
predict  the  attribute  levels.  Alternatively,  the  attribute  levels  may  be 
estimated  subjectively  or  established  from  performance  histories.  Use  of 
mappings  from  predictive  features  is  more  attractive  than  subjective 
estimates  as  no  load  is  imposed  on  the  operator,  and  situation-specific 
factors  may  be  taken  into  account.  For  example,  the  communication  delay 
may  be  directly  predicted  from  sensor  queue  length,  sensor  response 
characteristics,  and  transmission  distance.  Subjective  estimates  or 
pre-established  values  for  the  attribute  levels  would  tend  to  be  much  less 
reliable  than  such  in-task  calculations. 

2.3.4  Attribute  Weight  Estimation 

The  policy  defining  factors  in  the  model,  the  importance  weights 
k^ , are  parameters  suitable  for  either  objective  or  subjective  estimation. 

If  the  consequences  can  be  defined  along  objective  scales  (dollars,  ship- 
equivalents,  etc.),  then  the  weights  could  be  derived  by  analysis  and  input 
prior  to  system  operation.  Unfortunately,  Felson  (1975)  states  that  only 
in  a few  highly  structured  situations  can  such  an  optimal  model  be  derived. 
More  often,  the  operator's  goal  structure, expressed  as  importance  weights, 
must  be  elicited  or  inferred  and  then  incorporated  in  the  model.  There  are 
a number  of  advantages  to  such  subjective  estimation,  particularly  with 
respect  to  allocation  of  function.  By  incorporating  individualized  operator 
weights  in  the  model,  the  complex  evaluation  and  goal  direction  functions 
remain  the  responsibility  of  the  operator,  while  the  normative  aggregation 
functions  are  assumed  by  the  computer.  Also,  operator  acceptance  of  aiding 
by  the  model  may  be  increased  since  his  preferences  are  incorporated  in  the 
machine  decisions. 

The  operator's  subjective  weights  may  be  defined  off-line  by 
elicitation  or  on-line  through  inference.  The  off-line  methods  include 
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direct  elicitation  of  preference,  decomposition  of  complex  gambles  into 
hypothetical  lotteries,  and  use  of  multi-variate  methods  to  analyze  binary 
preference  expressions.  These  techniques  are  accurate  and  reliable  in 
many  circumstances,  but  they  have  a number  of  disadvantages  when  applied 
to  operational  systems.  Typically,  these  methods  require  two  separate 
stages  --  assessment  and  application.  Assessment  requires  an  interruption 
of  the  task  and  elicitation  of  responses  to  hypothetical  choices.  Problems 
arise  with  such  procedures  since  the  operator's  judgments  may  not  transfer 
to  the  actual  situation;  the  decision  maker  may  not  be  able  to  accurately 
verbalize  his  preference  structure  (Macrimmon  and  Taylor,  1972);  and  the 
judgments  made  in  multi-dimensional  choices  are  typically  responses  to 
non-general izable  extreme  values  (Keeney  and  Sicherman,  1975). 

Estimation  techniques  relying  on  inference  from  in-task  behavior  may 
be  more  useful.  The  inference  techniques  can  be  based  on  non-parametric 
forms  of  pattern  recognition.  Here  a model  of  decision  behavior  is  assumed 
and  the  parameters  of  the  model  are  then  fitted  by  observation  and 
adjustment.  Briefly,  the  technique  considers  the  decision  maker  to  respond 
to  the  characteristics  of  the  various  alternatives  as  patterns,  classifying 
them  according  to  preference.  A linear  discriminant  function  is  used  to 
predict  the  decision  maker's  choices,  and  when  amiss,  is  adjusted  using  error 
correcting  procedures.  In  this  way,  no  preference  ratings  or  complex 
hypothetical  judgments  are  required  of  the  operator. 

The  adaptive  nature  of  the  estimation  program  is  shown  in  Figure 
2-2.  Expected  consequence  vectors  associated  with  each  information  source 
are  input  to  the  model.  These  consequence  vectors  are  dotted  with  the 
weight  vector,  resulting  in  evaluations  along  a single  utility  scale.  The 
maximum  utility  choice  is  determined  and  compared  with  the  operator's 
actual  choice.  If  a discrepancy  occurs,  the  weight  vector  is  adjusted 
according  to  the  following  rule: 
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FIGURE  2-2.  ADAPTIVE  ESTIMATION  PROCESS 
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where 


Ik'  is  the  updated  weight  vector 
k is  the  previous  weight  vector 
x is  an  adjustment  constant 

x is  the  attribute  vector  of  the  chosen  alternative 
— c 

x is  the  mean  attribute  vector  of  all  alternatives  ranked  by  the 
-m 

model  above  the  chosen  alternative. 


Ideally,  the  error  correction  moves  the  weight  vector  in  a direction 
minimizing  subsequent  errors.  The  amount  of  movement  depends  on  A,  the 
adjustment  increment.  Nilsson,  1965,  describes  several  different  forms  of 
x that  can  be  used  depending  on  the  combination  of  speed  and  smoothing 
desired. 


The  type  of  criteria  used  for  model  training  is  also  a major 
consideration.  The  training  may  be  based  on  objective  outcomes  such  as 
stock  market  consequences,  or  subjective  criteria  such  as  actual  operator 
decisions,  or  on  some  combination  of  subjective  and  objective  criteria.  The 
approach  based  on  both  objective  and  subjective  criteria  is  the  most 
involved.  In  many  situations,  an  occasional  indicator  of  objective 
performance  is  observable.  The  aircraft  may  be  lost,  the  target  attained, 
or  some  number  of  subgoals  accomplished.  In  this  way,  the  correctness 
of  a sequence  of  actions  may  become  objectively  known.  The  utility  model 
would  be  trained  subjectively  prior  to  this  by  observation  of  the  operator's 
choices.  If  the  sequence  of  choices  led  to  an  objectively  favorable  outcome, 
the  trained  parameter  set  would  be  retained.  If  the  outcome  was  unfavorable, 
the  parameter  set  would  be  returned  to  the  levels  present  prior  to  the 
sequence  of  decisons.  In  this  way,  objective  criteria  would  guide  overall 
training,  but  the  explicit  decision-by-decision  policy  for  information 
management  would  be  subjectively  derived. 


Of  course,  the  adaptive  techniques  of  estimation  described  above 
are  warranted  only  if  repetitive  decisions  are  available  for  training  and 
if  the  differential  weighting  of  attributes  is  important.  In  cases  where 
only  a few  non-repetitive  decisions  will  be  made,  off-line  estimates  of 
the  weights  k.  are  favored.  Here,  techniques  such  as  direct  estimates, 
hypothetical  lotteries,  or  multiple  regression  are  used  for  estimation 
prior  to  the  mission.  It  is  assumed  with  these  techniques  that  the  system 
requirements  will  not  change  after  the  estimates. 

Questions  concerning  the  importance  of  differential  weighting  are 
more  basic.  Unit  weighting  schemes  (in  which  all  weights  k..  are  set  equal 
to  1.0)  have  been  found  to  be  quite  effective  in  certain  circumstances. 
Errors  in  the  model  form,  positive  correlations  between  variables,  and 
small  sample  sizes  all  reduce  the  predictive  capabilities  of  differential 
weights  compared  to  unit  weights  (Einhorn  and  Hogarth,  1975).  Essentially, 
the  more  precise  and  parsimonious  the  model,  the  more  important  differential 
weights  are. 

Unit  weighting  schemes  are  expected  to  see  only  minor  application  in 
aiding  advanced  aircraft  operations.  Careful  selection  of  attributes 
minimizes  intercorrelations  between  variables,  and  the  correlations  that  do 
occur  should  tend  to  be  negative.  For  example,  in  most  cases  costly 
information  is  generally  more  informative  than  inexpensive  information, 
and  equipment  attrition  tends  to  be  negatively  correlated  with  goal 
attainment.  These  circumstances  favor  inferred  weight  models.  Unit 
weighting  schemes  should  primarily  be  useful  as  starting  points  for 
estimation,  or  as  strategies  for  situations  in  which  a great  deal  of  noise 
is  present. 

The  processes  described  above  potentially  form  the  basis  of  a 
decision  aiding  system.  The  sequence  can  be  divided  into  three  segments  -- 


modeling,  analysis,  and  execution.  Modeling  consists  of  structuring,  the 
definition  of  the  various  components  of  the  decision  model,  and  assessment, 
the  determination  of  the  parameter  levels.  These  operations  are  normally 
shared  in  function.  The  human  defines  the  problem  structuring  (at  least 
until  self-organizing  systems  are  realized)  and  the  computer  performs  the 
assessment.  The  second  segment,  analysis,  may  be  assigned  completely  to 
the  computer.  This  stage  involves  solving  the  model  to  determine  its 
implications,  and  computing  the  effects  of  altering  model  assumptions.  The 
final  segment,  execution,  is  again  a flexible  function:  either  man  or 
machine  may  make  the  decision.  In  the  early  stages  of  model  training,  the 
human  would  be  expected  to  perform  the  action  with  the  machine  observing 
passively.  Later,  with  increased  confidence  in  its  choices,  the  computer 
could  either  make  recommendations  to  the  operator  or  take  over  the  decision 
functions  autonomously  (subject  to  operator  override). 

Evidence  for  the  usefulness  of  the  multi-attribute  utility 
formulation  and  adaptive  estimation  programs  was  obtained  during  the  initial 
year  of  the  program  (Steeb,  Chen  and  Freedy,  1977).  A simulation  resembling 
control  of  a remotely  piloted  vehicle  (RPV)  was  used  in  this  study. 
Individual  subjects  navigated  the  RPV  through  a changing  hazardous 
environment.  In  doing  so,  the  operators  selected  different  combinations  of 
information  and  control  allocation.  The  adaptive  model  was  found  to  be  more 
predictive  of  subject's  behavior  than  either  a constant,  unity  weight  model 
or  an  off-line  method  of  weight  estimation.  Also,  the  model  was  found  to 
be  useful  in  identifying  different  decision  policies  or  styles. 

2.4  The  Adaptive  Model  as  a Decision  Aid 

2.4.1  Behavioral  Forms  of  Aiding.  The  adaptive  decision  model  has  the 
potential  of  improving  system  performance  through  the  following  three  means: 


(1)  Aggregation:  The  model  keeps  track  of  large  amounts  of  data, 
performs  probability  update  calculations,  and  computes  tree 
fold-back  values. 

(2)  Smoothing:  The  model  reduces  the  random  noise  or  error 
implicit  in  human  response.  This  smoothing  results  from  the 
averaging  effect  of  estimation. 

(3)  Augmentation:  Display  of  recommendations  by  the  adaptive  model 
may  serve  to  increase  the  operator's  decision  making  power. 
Observing  the  model  recommendations,  the  operator  may 
systematically  refine  his  behavior  and  possibly  consider  a 
larger  set  of  factors. 

The  aggregation  or  bookkeeping  functions  of  the  computer  are  an 
obvious  aiding  device.  The  decision  program  systematical ly  gathers  and 
processes  information  regarding  alternatives,  outcomes,  event  probabilities, 
and  objectives.  As  such  it  places  a rational  framework  on  operator  choices. 

Smoothing  or  reduction  of  random  effects  in  subjective  weighting  of 
data  is  a well-established  advantage  of  linear  models.  Linear  models  based 
on  an  operator's  average  behavior  typically  outperform  the  actual  behavior 
of  the  operator  (Bowman,  1963;  Goldberg,  1970;  Dawes  and  Corrigan,  1974). 
Aiding  by  model  recommendation  of  choices  and  by  model-based  automation 
should  result  in  this  type  of  performance  enhancement. 

The  third  area  of  improvement  provided  by  the  model,  augmentation, 
deals  with  sub-optimal  decision  behavior  that  is  more  deep-seated  than  noise 
or  random  effects.  Because  of  cognitive  limitations,  the  operator  can 
consider  only  a small  number  of  attributes  in  a decision.  In  complex 
situations,  he  then  constructs  his  own  simplified  and  manageable  model  of 


the  problem.  This  is  Simon's  (1957)  "principle  of  bounded  rationality" 
in  which  the  man's  behavior  may  be  consistent  with  his  own  simplified  model 
even  though  not  even  approximately  optimal  with  respect  to  the  real  world. 

The  sub-optimal  behavior  resulting  from  cognitive  limitations  may 
possibly  be  reduced  through  model-based  aiding.  Macrimmon  (1973)  suggested 
that  by  operating  in  parallel  with  the  DM,  a model  can  present  decision 
recommendations  based  on  a normative  processing  of  the  circumstances  and 
utilities.  The  operator's  task  is  then  changed  to  one  of  evaluation  and 
correction.  Freedy  and  his  associates  (1976)  displayed  such  model-based 
recommendations  to  operators  in  a simulated  task  of  submarine  surveillance. 
Significant  improvements  in  performance  resulted,  possibly  from  the 
opportunity  to  consider  more  complex  and  effective  strategies. 

Unfortunately,  the  parallel,  closed  loop  relationship  of  man  and 
model  engenders  some  problems  of  dynamics.  With  aiding,  the  decision 
faced  by  the  operator  includes  both  the  attribute  patterns  of  the  choices 
and  a normative  processing  of  those  patterns.  Since  this  processing  is 
based  on  his  previously  observed  behavior,  it  should  lead  to  greater 
consistency,  speed,  and  effectiveness  in  recurrent  situations.  However, 
it  may  result  in  inappropriate  recommendations  in  completely  new 
circumstances.  These  characteristics  are  typical  of  predictive  displays. 
The  predictions  are  only  accurate  if  future  behavior  can  be  estimated  from 
previous  observations.  Thus  with  a major  structural  change  in  the 
environment,  the  recommendations  may  be  based  on  irrelevant  data,  and  could 
slow  the  operator's  adjustment.  Kunreuther  (1969)  states  that  this  type 
of  lag  can  be  minimized  by  including  only  recent  decisions  or  by 
exponentially  weighting  the  observations  according  to  the  age.  A recency 
bias  of  this  type  is  realized  to  some  extent  by  virtue  of  the  adjustment 
mechanism.  An  additional  bias  may  be  necessary  in  rapidly  changing 
situations. 
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2.4.2  Aiding  Functions.  The  specific  forms  of  aiding  provided  by  the 
adaptive  model  may  be  classified  into  three  main  functional  areas: 

Problem  recognition,  situation  diagnosis,  and  option  selection.  The 
possible  contributions  to  each  of  these  areas  are  listed  below. 

Problem  Recognition.  This  aspect  of  the  decison  process  involves 
the  monitoring  of  an  ongoing  action  on  a set  of  descriptive 
dimensions,  comparing  the  set  against  acceptable  limits,  and 
determining  that  the  action  is  still  appropriate  to  the  situation. 
Outcomes  of  these  processes  either  initiate  a new  action,  or  modify 
an  ongoing  action. 

Problem  recognition  is  a key  function  in  airborne  tactical 
operations.  Aircraft  may  be  programmed  to  follow  a course  unless 
special  circumstances  arise.  Response  maneuvers  or  transfer  of 
control  to  the  supervisory  operator  may  be  specified  by  the  multi- 
attribute decision  model.  Again,  trade-offs  between  communication 
costs,  potential  losses,  operator  loading,  and  other  factors  are 
made  by  the  model.  This  function  unburdens  the  operator  of 
repeatedly  interrogating  the  system  as  to  its  status.  Also, 
available  empirical  evidence  suggest  that  men  tend  to  err  on  the 
side  of  conservatism  in  problem  recognition.  For  example,  Vaughan, 
Virnelson,  and  Franklin  (1964)  had  experienced  Army  officers 
monitor  a series  of  messages  that  indicated  the  need  to  change  the 
axis  of  advance  in  a simulated  attack  scenario.  With  only  one 
exception,  officers  did  not  modify  the  ongoing  action  plan  nor  did 
they  anticipate  the  possibility  of  changing  the  plan  in  spite  of  a 
series  of  messages  indicating  this  need  with  increasing  urgency. 
Thus,  in  some  circumstances  a automated  system  for  problem 
recognition  may  be  essential. 
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Situation  Diagnosis.  For  effective  decision  making  and  control 
functions,  the  operator  must  accurately  assess  and  integrate  the 
probability  of  occurrance  of  events.  Among  these  estimates  are 
judgments  regarding  likelihoods  of  obstacles,  detection  of 
communications,  possibility  of  visibility  loss,  etc.  Man  tends 
to  be  a weak  diagnostician  when  dealing  with  multiple  probabilities. 
Summarizing  results  from  several  studies  of  clinical  diagnosis, 
Goldberg  (1968)  concluded  that  diagnostic  judgments  are: 

(1)  unrel iable  overtime, 

(2)  unreliable  across  diagnosticians, 

(3)  only  marginally  related  either  to  experience  of  the  man  or 
to  his  confidence  in  the  accuracy  of  his  judgments, 

(4)  only  slightly  affected  by  amount  of  available  information, 

(5)  generally  of  low  validity. 

In  a similar  vein,  Edwards  (1963)  presented  evidence  from  non- 
clinical  studies  that  man  is  a relatively  good  probability  estimator 
for  single  items,  but  poor  at  aggregating  a number  of  probability 
estimates  to  form  a conclusion.  Additional  evidence  and  discussion 
of  this  misaggregation  effect  are  provided. by  Slovic  and  Lichtenstein 
(1971)  and  Rapoport  and  Wallsten  (1972). 

The  decision  model  performs  many  of  the  probability  aggregation 
functions  normally  required  of  the  operator  --  frequency  tabulations, 
probability  update  calculations,  etc.  The  program  performs  these 
calculations  more  rapidly,  reliably  and  accurately  than  is  possible 
manually.  The  operator  of  course,  may  still  provide  inputs  for  the 
probability  programs  in  the  form  of  single  item  prior  and 
conditional  probabilities. 
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Option  Selection.  The  options  open  to  the  decision  maker  are 
information  to  acquire  and  actions  to  select.  These  options  are 
interdependent,  as  noted  earlier  in  section  2-2.  The  value  of 
information  is  due  to  its  impact  on  the  subsequent  action  decison. 
Each  of  these  choices,  information  acquisition  and  action  selection, 
is  tied  to  maximization  of  overall  expected  gain  or  utility. 

For  the  operator,  simultaneous  consideration  of  multiple  alternatives 
portrayed  against  multiple  criteria  quickly  becomes  too  complex  for 
easy  resolution.  For  example,  Connolly  and  his  associates  conducted 
a series  of  experiments  at  Hanscomb  Field  to  assess  the 
appropriateness  of  weapon  selection  decisions  by  experienced  Air 
Force  officers  in  a simulated  air  defense  environment  (Connelly, 

Fox  and  McGoldrick,  1961;  Connolly,  McGoldrick,  and  Fox,  1961). 
Although  instructed  to  use  three  criteria  in  the  selection  of 
weapons  to  targets  (minimize  damage  to  defended  area,  destroy 
maximum  number  of  threatening  objects,  and  conserve  counter-weapons), 
actual  selection  reflected  a disproportionate  weighting  of  the  three 
factors.  Again,  computer-based  aggregation  of  factors  is  indicated. 
The  coming  sections  describe  the  means  by  which  the  forms  of  aiding 
described  here  can  be  applied  to  the  problems  of  information 
selection  in  tactical  airborne  operations. 
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3.  DECISION  AIDING  IN  ADVANCED  AIRCRAFT  OPERATIONS 

3.1  General 

The  selection  and  presentation  of  information  in  tactical  airborne 
systems  introduces  some  special  problems  that  compound  those  mentioned 
earlier.  The  operator  must,  under  considerable  time  stress,  weigh  the 
probable  usefulness  and  costs  of  a variety  of  competing  forms  of  information 
--  mission  status,  track  data,  environmental  information,  aerodynamic 
functioning,  etc.  These  moment-to-moment  judgments  must  often  be  based  on 
subjective  factors,  since  the  decision  is  normally  too  complex  and  dynamic 
to  be  analytically  tractable. 

The  adaptive  programs  described  earlier  hold  a great  deal  of 
promise  for  aiding  in  advanced  aircraft  operations.  Tactical  airborne 
systems  typically  require  consideration  of  voluminous  amounts  of  relevant 
and  irrelevant  information.  At  the  same  time,  severe  time  stresses  and 
interfering  tasks  can  degrade  the  information  integration  processes.  To 
reduce  the  operator  load,  combat  aircraft  typically  have  sophisticated 
autopilot  and  weapon  control  systems.  These  systems  are  expected  to  be 
extendable  to  include  functions  of  information  evaluation  and  management, 
since  they  take  into  account  the  key  factors  of  environmental  state,  vehicle 
state,  and  task  objectives.  The  following  sections  describe  the  definition, 
structuring,  and  implementation  of  adaptive  decision  models  in  the  context 
of  tactical  airborne  operations. 

3.2  Information  System  Characteristics 

The  typical  advanced  aircraft  mission  can  be  defined  by  a series  of 
mission  phases,  much  like  the  stages  of  the  RPV  supervision  task  analyzed  in 
the  initial  year  (Steeb,  Chen  and  Freedy,  1977).  The  phases  can  be 
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characterized  by  the  danger  or  frequency  of  threats,  the  time  available 
for  decision  making,  and  the  options  and  characteristics  of  information 
concerning  the  aircraft  and  the  environment. 

The  information  available  at  a given  time  is  dependent  on  the 
environmental  situation,  the  sensor  characteristics,  the  data  base  content, 
and  the  display  capabilities.  The  information  itself  may  consist  of  data 
regarding  weather  conditions,  aerodynamic  status,  target  track,  ECM,  and 
mission  status. 

The  costs  of  acquiring  information  result  from  the  sensor 
characteristics,  the  direct  and  indirect  costs  of  sensor  deployment, 
information  processing  and  display,  and  the  amount  of  attention  the 
operator  can  contribute.  The  direct  costs  of  information  acquisition 
include  such  factors  as  energy  expenditures  and  equipment  expenses. 

Indirect  costs  include  increased  possibilities  of  detection  and 
countermeasures.  The  available  operator  attention,  finally,  is  defined 
by  the  task  demands  and  the  individual  capabilities  of  the  operators. 

The  costs  and  payoffs  associated  with  the  various  possible  outcomes 
vary  with  mission  phase.  The  consequences  are  defined  not  only  in  terms 
of  attrition  of  equipment  and  attainment  of  objectives,  but  also  as  a 
function  of  organizational  policy  and  procedures.  The  relative  importance 
of  fuel  expenditures,  vehicle  survival,  countermeasures,  etc.,  change  as 
the  mission  objective  is  approached,  attained,  or  past.  The  relative 
importance  of  these  factors  must  be  assigned  by  the  human  operator  or  by 
the  coirriand  group. 

Available  time  for  decision  making  varies  throughout  the  mission 
as  a direct  function  of  the  varying  vehicle  speed,  altitude,  and  surrounding 
weather  conditions.  Altitude,  cloud  cover  and  ECM  determine  the  distance 


that  obstacles,  navigation  points,  or  targets  can  be  observed.  The  speed 
then  determines  the  available  time.  Decision  time  can  be  expected  to 
influence  the  amount  of  information  that  can  be  processed  and  the 
probability  distribution  of  the  possible  consequences. 

In  sum,  the  selection  of  information  and  control  to  allocate  to 
the  supervisory  human  operator  is  a complex  and  dynamic  decision.  The 
decision  maker  must  continually  weight  the  probable  usefulness  of  the 
information  against  the  costs  of  acquiring  it.  Since  this  decision  is 
especially  difficult,  the  costs  and  benefits  are  both  mul ti -dimensional 
and  probabilistic. 

3.3  Structure  of  the  Information  Seeking  Model 

3.3.1  Information  Management  Functions . The  major  information  management 
functions  faced  by  the  operator  are  diagramed  schematically  in  Figure  3-1. 
The  information  available  consists  of  data  regarding  the  aircraft,  the 
targets,  the  environment,  and  operator  and  system  capabi 1 i ties . The 
information  is  then  used  by  the  operator  to  perform  supervisory  control 
actions. 


The  information  and  control  choice  sequence  is  that  diagramed 
earlier  with  the  decision  tree  of  Figure  2-1.  This  diagram  is  repeated  with 
labels  representative  of  tactical  airborne  operators  in  Figure  3-2.  The 
mul ti-attribute  utility  formulation  provides  a useful  basis  for  structuring 
both  the  information  and  control  decisons.  The  specific  steps  of  the 
modeling  process  are  outlined  in  Figure  2-3.  The  figure  shows  the  two 
sides  of  the  modeling  problem,  probability  estimation  and  utility  assessment. 
The  upper  portion  of  the  figure  details  the  processes  cf  probability 
estimation.  These  include  delineation  of  the  possible  states  of  the 
environment,  evaluating  the  current  level  of  uncertainty  concerning  states. 
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FIGURE  3-1:  MAJOR  INFORMATION  MANAGEMENT  FUNCTIONS 
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selecting  information  to  reduce  the  uncertainty,  and  revising  the 
probability  estimates  in  light  of  the  new  data.  The  lower  portion  of  the 
figure  is  concerned  with  outcome  evaluation  or  utility  estimation.  Here 
the  levels  and  importance  weights  for  each  dimension  of  consequence  are 
determined. 


The  key  element  in  the  probability  estimation  sequence  is  the 
information  acquisition  stage  (enclosed  by  dotted  lines).  Figure  3-4 
elaborates  this  stage,  showing  the  steps  that  go  into  the  choice  of 
information  and  the  subsequent  incorporation  of  the  datum  into  the 
situation  estimate.  The  upper  portion  of  the  figure  deals  with  the 
information  source  selection.  The  characteristics  of  the  various  available 
sources  are  determined  by  observation  and  analysis.  This  estimation  of  the 
characteristics  of  the  information  sources  is  accomplished  by  successive 
comparisons  of  messages  received  and  subsequently  observed  states.  The 
choice  of  information  source  is  then  made  according  to  the  potential  impact 
of  the  information  on  the  prior  probability  estimate.  Once  a source  is 
selected  and  a datum  observed,  the  information  is  incorporated  into  a 
revised  situation  estimate  through  Bai-^'  rule: 


p(zhiV 


P(yj|zh>  • PUh> 

HSJT 


(3-1) 


where  P(y^)  = J P(yj|zh)  • P(zh) 


P(zh|yj)  is  the  probability  of  state  zh  being  present  given  that  message  y^ 
was  received. 

The  other  major  modeling  process  is  utility  assessment  or  outcome 
evaluation.  The  possible  combinations  of  actions  and  states  are  enumerated 
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FIGURE  3-4:  PROCESSES  INVOLVED  IN  PROBABILITY  ESTIMATION 


(This  is  an  elaboration  of  the  "Information 
Acquisition"  block  of  Figure  3-3) 
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off-line  prior  to  a mission.  The  problem  is  then  to  assign  consequence 
levels  and  importance  weights  along  a predefined  set  of  dimensions.  Figure 
3-5  elaborates  this  process.  The  first  step  is  the  selection  of  an 
independent,  exhaustive,  and  predictive  attribute  set.  The  attributes  are 
the  various  constituent  aspects  of  the  consequences.  Each  combination 
of  information,  action  and  outcome  is  associated  with  a set  of  attribute 
levels.  This  is  done  by  observation  and  adjustment,  just  as  in  the 
determination  of  information  source  characteristics.  Scaling  procedures 
are  applied  to  the  raw  consequence  dimensions  to  arrive  at  normalized 
values.  Each  attribute  is  scaled  so  that  its  plausible  range  spans  zero  to 
one.  These  processes  result  in  a specification  of  the  parameters  of  the 
basic  multi-attribute  formulation: 


M 


N 


E tu(x)]$  = I P(zk)  l K.  uk  (x  ) 


(3-2) 


K=1 


i=l 


where 


E [u(x)]$  is  the  expected  utility  of  information  choice  s. 

P(zk)  is  the  probability  of  state  k with  this  information  choice 
K.  is  the  importance  weight  for  attribute  i,  and 

x..jk  is  the  level  of  attribute  i associated  with  action  j and  state  k 

The  following  sections  will  develop  some  of  the  specifics  of  the 
modeling  cycle. 
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FIGURE  3-5:  PROCESSES  INVOLVED  IN  OUTCOME  EVALUATION 


(This  is  an  elaboration  of  the  "Outcome 
Evaluation"  block  of  Figure  3-3) 
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3.3.2  Attribute  Development.  The  attributes  of  an  information  seeking 
decision  are  dimensions  of  consequence  that  are  common  to  all  tips  of  the 
decision  tree  (shown  earlier  in  Figure  3-2).  These  dimensions  may  include 
communications  costs,  equipment  losses,  goal  attainments,  future  effects, 
and  other  factors.  In  the  end,  the  constituent  effects  will  be  weighted 
and  aggregated  together  to  arrive  at  an  overall  evaluation  of  an  outcome. 


The  actual  choice  of  the  attribute  set  is  extremely  important. 

Dawes  (1975)  states  that  the  choice  of  factors  to  include  is  probably 
of  greater  impact  than  the  determination  of  the  model  form.  Desirable 
characteristics  are  accessibility  for  measurement,  independence, 
monotonicity  with  preference,  completeness  of  the  set,  and  meaningful  ness 
for  feedback.  Monotonicity,  in  this  content,  implies  that  an  increase  in 
the  attribute  level  always  results  in  an  increase  in  preference.  If  the 
attribute  levels  are  monotonic,  a simplification  is  possible.  (Fisher 
(1972)  and  Gardiner  (1974)  note  that  a straight  line  approximation  to  the 
utility  function  results  in  minor  losses  of  model  accuracy.  The  estimated 
utility  (ignoring  uncertainty  for  now)  is  then  a weighted  linear  combination 
of  attribute  levels: 


U(a  ,z,  ) = Y k.x. 
z h v 


hk 


(3-3) 


where  U(a  .z,  ) is  the  utility  of  state  h occurring  with  action  z,  k.  is  the 
z h i 

importance  weight  for  attribute  i,  and  x.^  is  the  level  of  attribute  i 
associated  with  state  h and  action  k. 

Information  costs  may  comprise  attributes  of  special  note.  Often, 
the  benefits  of  an  information  acquisition  are  simply  weighted  against  the 
costs  of  acquiring  the  information.  If  a net  gain  is  anticipated, 
acquisition  of  the  information  is  considered  justified.  Often,  though  * 
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the  costs  themselves  are  multidimensional,  comprising  energy  costs,  time 
delays,  equipment  expenditures,  and  risks  of  detection.  The  scaling, 
weighting,  and  aggregating  of  these  costs  may  be  most  easily  performed  in 
combination  with  all  of  the  non-cost  attributes  --  tactical  gains,  political 
impact,  etc.  Then,  trade-offs  among  each  of  the  factors  may  be  performed 
in  a single,  consistent  operation. 

A candidate  set  of  attributes  might  contain  factors  from  five  areas: 

(1)  Communications  Costs  - The  expenditures  associated  with  use 

of  the  information  sources.  These  may  include  requirements  of 
energy,  equipment,  and  operator  attention. 

(2)  Equipment  Attrition  - Consequences  concerning  the  integrity 

of  the  vehicle.  Included  are  fuel  expenditures,  system  damage, 
and  vehicle  loss. 

(3)  Objective  Attainment  - The  degree  of  accomplishment  of  the 
mission  objectives.  Target  goals  may  be  the  area  reconnoitered, 
adversaries  dispatched,  and  political  impact  obtained. 

(4)  Dynamic  Effects  - The  future  consequences  resulting  from  the 
current  actions.  Tnese  consequences  may  include  effects  on 
subsequent  action  choices,  availability  of  future  information, 
and  changes  in  the  environment  resulting  from  the  action. 

(5)  Subjective  Needs  - The  operator  may  have  propensities  for 
obtaining  (or  refusing)  information  beyond  that  called  for  by 
the  above  factors.  These  preferences  relect  needs  of  task 
continuity,  maintenance  of  load,  or  other  idiosyncratic  factors. 
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A useful  consequence  set  might  contain  a single  dimension  or 
attribute  from  each  of  these  categories.  In  fact,  five  attributes  appears 
to  be  an  upper  limit  to  the  number  of  factors  a decision  maker  can 

effectively  consider  (V.  Winterfeldt,  1975).  If  several  factors  contribute 

to  one  consequence  dimension,  these  factors  should  be  combined  using  a 
single  common  scale  --  dollars,  ship-equivalents,  fuel  quantity,  etc. 

Each  of  the  attributes  --  communications  costs  vehicle  losses, 
etc.,  --  must  be  scaled  with  interval  properties  along  e set  range.  The 
least  desirable  consequence  that  may  occur  is  assigned  a level  of  zero  on 

the  scale.  The  most  desirable  consequence  is  assigned  a level  of  one.  The 

weighting  factors  should  also  be  normalized  so  that  the  overall  worst 
combination  of  factors  results  in  a value  of  zero  and  the  overall  best 
combination  a value  of  one. 

A special  situation  occurs  with  probabilistic  attributes.  Assuming 
risk  neutrality,  probabilistic  consequences  may  be  computed  according  to 
their  expected  level.  For  example,  the  vehicle  loss  attribute  may  have 
three  possible  levels,  each  with  a different  probability  of  occurrence. 

The  expected  value  is  computed  by  the  following  additive  expression: 

3 

E ( x . .)  = y P(z, )x.  ..  (3-4) 

i j 7 L k'  ijk  ' 

k=l 


where  the  parameters  are  defined  as  in  Equation  3-3.  Once  the  attributes  are 
defined  and  their  levels  are  determined,  the  aggregation  rule  must  be 
identified.  The  attributes  - costs,  losses,  delays,  future  impacts,  etc., 

- may  combine  in  an  additive,  multiplicative,  or  more  complex  fashion  (see 
Keeney  and  Raiffa,  1975,  for  a description  of  some  of  the  more  popular 
formulations).  For  the  work  here,  the  simple  additive  form,  exemplified  by 
Equation  3-3  appears  to  be  the  most  suitable.  The  additive  form  is  robust. 
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intuitively  easy  to  understand,  and  simple.  Also,  the  linear  form  of  the 
additive  will  be  seen  to  be  amenable  to  estimation  by  pattern  recognition 
techniques. 


3.3.3  Consequence  Level  Determination.  The  actual  level  of  each  of  the 
attributes  for  a given  outcome  can  be  determined  by  mappings  between 
predictive  features  and  the  attributes.  Predictive  features  must  be 
identified  which  are  accessible  to  an  onboard  program  and  capable  of 
determining  the  consequence  levels.  Mappings  between  the  predictive 
features  and  the  attributes  are  either  pre-established  or  determined  by 
observation  and  adjustment. 

The  data  available  to  the  decision  program  are: 

(1)  Directly  sensed  information  concerning  the  environmental 
state  (weather,  terrain,  ECM,  target  track). 

(2)  The  vehicle  state  (velocity,  fuel,  autopilot  capability). 

(3)  The  information  system  characteristics  (capacity,  noise, 
cost) . 

(4)  Tactical  data  (technical  characteristics  of  own  and  enemy 
aircraft,  sensors  and  weapons;  information  about  the  operations 
area) . 

(5)  Action  alternatives  (control  responses,  weapon  deployment). 

(6)  Operator  capabilities  (attention,  load). 
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A manageable  subject  of  these  features  must  be  determined.  The 
consequence  mapping  can  then  be  refined  by  comparison  of  the  predicted  and 
actually  observed  consequences.  The  mapping  can  be  developed  either  by 
prior  definition,  by  regression,  or  by  the  pattern  recognition  techniques 
described  in  the  coming  section. 

3.3.4  Weight  Assessment.  The  method  of  assessment  developed  in  the 
initial  year's  study  --  adaptive  estimation  using  pattern  recognition  -- 
is  well  suited  to  the  information  evaluation  and  management  problem.  The 
goal  is  to  estimate  the  operator's  decision  making  policy  by  observation  of 
his  choices.  The  procedure  was  diagramed  earlier  in  Figure  2-2.  First, 
expected  consequence  vectors  associated  with  each  information  choice  are 
input  to  the  model.  These  consequence  vectors  are  dotted  with  the  weight 
vector,  resulting  in  evaluations  along  a single  scale.  The  maximum  expected 
utility  choice  is  determined  and  compared  with  the  operator's  actual  choice. 
If  a discrepancy  occurs,  the  weight  vector  is  adjusted  according  to  the 
procedure  represented  by  Equation  2-4.  Ideally,  the  error  correction  moves 
the  weight  vector  in  a direction  minimizing  subsequent  errors.  A detailed 
description  of  the  structure  and  dynamics  of  the  estimation  procedure  is 
given  in  our  previous  technical  report  (Steeb,  Chen  and  Freedy,  1977). 

An  alternative,  non-adaptive  means  of  weight  assessment  is  possible 
through  use  of  direct  elicitation.  The  operator  is  familarized  with  the 
task  situation  and  then  presented  with  a series  of  choices  such  as  that 
shown  in  Figure  3-6.  Here  one  attribute  (cost)  is  assigned  an  arbitrary 
weight  and  each  remaining  attribute  is  compared  to  it.  The  comparisons 
between  attributes  are  tied  to  the  possible  range  that  each  attribute  spans, 
hence  the  use  of  the  anchored,  worst-to-best  scales  shown  in  the  figure. 

The  estimation  procedure  is  as  follows:  If  a shift  in  cost  of  an  alternative 
from  16  points  to  no  cost  has  a value  of  10  (this  is  an  arbitrary  baseline), 
what  is  the  value  of  a shift  in  delay  of  6 seconds  to  zero  delay?  A value 
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of  20  would  mean  that  this  shift  in  delay  is  twice  as  important  as  the  cost 
shift,  a value  of  5,  half  as  important.  Similar  comparisons  are  made 
between  the  baseline  attribute  (cost)  and  each  remaining  attribute.  In  the 
end,  all  weights  are  normalized  so  as  to  sum  to  1.0.  This  procedure  is 
representative  of  many  direct  elicitation  approaches,  and  exhibits  some  of 
the  problems  associated  with  these  techniques  - hypothetical  choices,  use 
of  extreme  points,  interference  with  the  task,  etc.  Nevertheless,  it  is 
a useful  means  of  estimation  in  many  situations.  A experimental  comparison 
of  the  adaptive  technique  and  direct  elicitation  method  will  be  found  in 
Chapter  Four. 

3.3.5  Probability  Estimation.  The  major  probability  parameters  requiring 
estimation  are  the  prior  probabilities  P(z)  and  the  conditional  probabilities 
P(y|z).  The  priors  are  the  probabilities  of  state  z in  a particular 
situation.  The  conditional  probabilities  deal  with  the  likelihood  of 
receipt  of  message  y if  state  z is  present.  Both  of  these  forms  of 
probabilities  can  be  estimated  from  frequency  counts. 

A second  area  of  uncertainty  concerns  the  consequence  levels 
associated  with  a given  message  and  state.  These  are  the  performance 
probabilities  and  are  derived  from  stored  data:  Detection  range,  target 
hardness,  personnel  performance,  system  reliability,  guidance  system 
accuracy,  etc.  The  probability  of  outcome  given  the  message  received  can 
be  computed  for  each  set  of  actions.  Comparison  of  the  messages  received, 
actions  taken  and  the  consequences  subsequently  observed  provide  the 
necessary  data. 

3.3.6  Level  of  Detail.  The  level  of  detail  handled  by  the  model  is 
basically  a problem  of  efficiency  of  categorization.  Choices  of  the  fineness 
of  distinction  of  information  modes,  messages,  and  states  involves  a trade- 
off the  model  complexity  (and  processing  time)  against  degree  of  specification 
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of  aiding.  For  example,  the  information  available  may  be  classified  by 
context  element  (threat  identity,  weather  formation,  malfunction  location, 
terrain  characteristic,  etc.)  or  by  source  (radar,  infra-red  sensor,  video, 
tactical  data  base,  radio,  etc.).  Similiarly,  the  messages  and  states  may 
be  represented  in  the  model  as  specific  details  of  threat  position,  course, 
speed  and  identification,  or  more  globally  as  specific  threat  presence  or 
absence.  Of  course,  separate  prior  and  conditional  probability  estimates 
must  be  maintained  for  each  state  represented  in  the  model. 

Much  of  the  question  of  level  of  detail  was  to  do  with  the  concept 
of  payoff  relevance,  a term  introduced  by  Marschak  (1963).  The  partitioning 
of  the  information  space  must  result  in  differences  in  (1)  the  existing 
representation  of  the  decision  situation,  (2)  the  actual  decisions  made, 
and  (3)  the  utility  resulting  from  the  changed  decisions.  Information  may 
be  ineffective  in  changing  the  situational  representation  and  resulting 
decisions  because  the  data  is  too  coarse  or  too  fine.  Information  that  is 
too  coarse  fails  to  distinguish  between  effectively  different  states  of 
nature  for  at  least  one  of  the  alternative  actions.  Information  that  is 
too  fine  differentiates  between  states  having  identical  payoffs  for  all 
actions.  Effective  information  --  data  that  is  not  too  fine  or  too  coarse 
--  is  termed  by  Marschak  to  be  payoff  relevant.  In  the  experimental 
application  of  the  model,  an  attempt  will  be  made  to  structure  the  model  so 
that  the  cost  of  the  chosen  level  of  detail  is  commensurate  with  the 
benefits. 

3.4  Aiding  in  Advanced  Aircraft  Operations 

3.4.1  Forms  of  Aiding.  It  was  noted  earlier  that  three  major  forms  of 
aiding  are  possible  once  the  operator's  decision  policy  is  estimated:  (1) 
system  evaluation  and  design,  (2)  automated  management  of  information,  and 
(3)  operator  training.  This  section  will  describe  in  more  detail  the 
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potential  of  model-based  aiding  for  tactical  airborne  systems.  Recounting 
briefly,  system  evaluation  and  design  entails  the  determination  of  each 
system  component's  contribution  to  information  value  (satisfaction  of 
operator  needs).  Specification  of  system  configuration  commensurate  with 
task  demands  and  individual  operator  needs  is  then  possible.  Automated 
management  of  information  is  the  moment-by-moment  control  of  display  content 
and  format  by  selection  among  available  information.  Operator  training, 
finally,  involves  the  use  of  policy  feedback  to  train  the  operator  to  make 
effective  information  and  control  decisions.  The  first  two  of  these  uses 
of  the  decision  model,  system  evaluation  and  automated  information 
management,  will  be  described  in  greater  detail  in  the  following  sections. 
Model-based  training  is  presently  outside  the  scope  of  this  work. 

3.4.2  System  Evaluation.  Two  types  of  evaluation  are  possible  using  the 
information  extracted  from  the  information  value  model:  direct  contribution 
and  marginal  contribution.  Each  of  these  evaluation  measures  is  described 
below. 


Direct  Contribution.  This  is  the  user-specific  value  of  a given 
information  source  in  a given  task  situation.  As  such,  it  is  a simple 
aggregation  of  components,  weighted  by  the  user's  policy: 

i"f0  value  Jks  - l kfJ  xjk  (3-5) 

attributes 

i 


Where  info  value  jks  is  the  aggregate  value  of  source  s to  user  j in 

situation  k,  k..  is  the  importance  weight  of  attribute  i to  user  j;  and  k.. 

* J i j 

is  the  mean  level  of  attribute  i in  situation  k.  This  formulation  is  useful 
when  each  information  source  contributes  to  a different  task  - threat 
detection,  navigation,  etc.  The  direct  contribution  measure  does  not  deal 
with  information  sources  having  overlapping  function. 
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Marginal  Contribution.  In  a group  of  information  sources  with 
overlapping  function,  the  information  value  of  one  source  can  be  calculated 
with  the  following  expression: 


information  value 


jks  ^ kij 
attributes 
i 


max 

x.  - over 
remaining 
sources 


T k . . x .. 
L ij  ik 

i 


(3-6) 


This  is  the  incremental  value  of  a source  over  the  next  most  highly  valued 
source.  The  summation  of  all  positive  contributions  for  a given  source 
indicate  the  source's  value  in  the  particular  task  situation. 

Using  either  the  direct  or  marginal  measure  of  information  value, 
the  mission  value  of  an  information  source  can  be  calculated  as  the 
summation  across  the  probability  distribution  of  task  situations. 

info  value  . = 7 prob  (situation  k)  • information  value  .. 

Js  L jks  (3.7) 


This  provides  an  overall,  user-specific  index  of  information  value. 

3.4.3  Automated  Management  of  Information.  The  information  value  model 
described  in  section  3.3  can  be  used  directly  for  management  of  information. 
The  multi-attribute  utility  model  represents  the  policy  of  the  specific  user, 
it  has  access  to  the  factors  characterizing  each  information  choice,  and  it 
can  be  linked  to  the  onboard  information  control  system.  The  model  can  thus 
be  configured  to  automatically  scan  the  available  information  sources,  select 
the  immediately  most  useful  source,  and  display  it  to  the  operator.  The 
following  sections  describe  an  initial  application  of  such  an  aiding  program. 


3-20 


4.  EXPERIMENTAL  STUDY 

4.1  Overview 

An  experimental  study  was  performed  to  test  the  effectiveness 
of  the  adaptive  decision  model  in  information  management.  An  advanced 
aircraft  simulation  was  developed,  replacing  the  RPV  simulation  used 
in  the  initial  studies.  (Steeb,  Chen  and  Freedy,  1977.)  Individual 
subjects  were  required  to  pilot  a simulated  aircraft  in  a changing, 
hazardous  environment.  In  doing  so,  the  operators  were  able  to  select 
from  a variety  of  forms  of  information  concerning  the  multiple  threats 
encountered,  and  take  either  evasive  or  aggressive  actions.  Performance 
comparisons  were  made  in  the  study  between  adaptive  and  non-adaptive 
techniques  of  estimation  of  model  parameters. 

4.2  Hypotheses 

The  following  experimental  hypotheses  were  tested: 

(1)  The  multi-attribute  model  can  be  used  to  accurately 
evaluate  the  effectiveness  of  different  information 
system  configurations  under  a variety  of  task  conditions. 

(2)  Automated  management  of  information  by  the  adaptive  model 
results  in  performance  superior  to  that  obtained  with 
the  off-line  model  and  superior  to  that  obtained  with 
manually  selected  information. 

(3)  The  improvement  of  aiding  in  either  form  of  automated 
information  management  will  be  enhanced  in  conditions  of 
higher  speed  stress. 
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(4)  Operators  will  be  more  accepting  of  automated  information 
management  in  conditions  of  high  speed  stress  and  adaptive 
estimation. 

4.3  Task  Simulation 

The  task  simulation  is  patterned  after  an  important  and 
representative  information  acquisition  task  --  multiple  threat  intercept 
operations  in  advanced  aircraft.  The  simulation  is  an  adaptation  of  the 
remotely  piloted  vehicle  supervision  task  employed  in  the  previous  study. 
(Steeb,  Chen  and  Freedy,  1977.)  The  format  of  the  display  is  similar 
to  that  used  previously,  but  threats  replace  the  obstacles,  and  the 
actions  open  to  the  operator  are  more  varied,  encompassing  both  avoidance 
and  attack  options.  Also,  a wider  variety  of  information  sources  are 
available  to  the  operator.  Briefly,  the  simulation  requires  the  operator 
to  control  an  advanced  aircraft  in  a hazardous  mission.  Threats  of 
uncertain  capability  and  location  are  encountered  repeatedly.  The 
operator  has  the  option  of  accessing  several  forms  of  information  about 
the  threats.  The  forms  of  information  differ  in  threat  discrimination 
capabilities,  transmission  costs,  processing  delays,  and  potential  of 
detection. 

4.3.1  Displays  and  Controls.  The  simulation  uses  a computer-generated 
CRT  display,  illustrated  in  Figure  4-1.  The  environment  and  aircraft 
are  shown  as  they  would  be  in  a moving-map  display.  Sets  of  threats 
appear  at  random  positions  at  the  upper  edge  of  the  display  and  move 
downward  at  a constant  velocity.  The  operator  can  move  the  vehicle 
symbol  horizontally  to  one  of  eleven  different  pathways  to  avoid  the 
threats,  or  he  can  remain  on  course  and  take  an  aggressive  action  against 
one  of  the  threats.  The  actions  open  to  the  operator  are  primarily 
decision  making  in  nature.  Dynamics  of  control  are  minimized  since  the 
threat  and  vehicle  velocities  are  held  constant. 
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The  threats  introduce  both  uncertainty  and  danger  to  the  task 
simulation.  Each  type  of  threat  has  a region  of  possible  damage  to  the 
aircraft  as  shown  in  Figure  4-2.  The  probability  of  damage  is  a function 
of  the  horizontal  distance  between  the  threat  and  the  piloted  aircraft. 

For  ease  of  learning,  the  four  obstacle  types  are  designed  to  be  evocative 
of  the  types  of  contact  expected  to  occur  in  actual  flight  missions  -- 
missiles,  fixed-wing  aircraft,  helicopters,  and  false  alarms  (birds). 

4.3.2  Information  and  Action  Options.  The  identity  and  location  of  the 
threats  can  be  determined  only  by  exercising  one  of  the  five  information 
options.  The  options  differ  in  their  ability  to  differentiate  between 
and  to  locate  the  threats.  These  capabilities  are  enumerated  in  Table 
4-1.  In  those  situations  where  the  information  option  is  unable  to 
differentiate  between  threat  types,  a combination  symbol  is  displayed. 
Several  of  these  combination  symbols  are  shown  on  the  right  of  Table  4-1.. 

After  receipt  of  the  information,  the  operator  must  take  one  of 
the  following  actions: 

(1)  Avoidance  - route  the  aircraft  along  one  of  the  eleven 
pathways . 

(2)  Aggression  - continue  in  center  pathway,  and  fire  along  one 
of  the  eleven  pathways. 

Two  forms  of  outcomes  result  from  these  actions  --  losses 
sustained  and  tactical  gains.  Losses  result  from  damage  from  the  threats, 
while  tactical  gains  arise  from  neutralizing  the  threat.  The  amount  of 
gain  and  loss  depend  on  the  payoffs. 
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FIGURE  4-2:  THREAT  CHARACTERISTICS: 
HELICOPTER  (H) 

AIRCRAFT  (A) 

MISSLE  (M) 
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TABLE  4-1 

Information  Type 

Discrimination 

Location 

Symbols 

1. 

Full 

All 

All 

M,  A,  H.  B 

2. 

Out! ine 

All  but 

airplane/hel icopter 

All 

M,  AH,  B 

3. 

Biological 

Bird  only 

All 

MAH,  B 

4. 

Location 

None 

All 

X 

5. 

Left/right 

All 

Left/ri ght 

M,  A,  H,  B 

Sequence.  The  task  consists  of  a series  of  similar,  connected 
decisions.  Prior  to  the  appearance  of  any  threats,  the  operator  is 
appraised  of  the  circumstances  surrounding  the  upcoming  decision.  He 
must  then  make  an  information  selection  by  pressing  one  of  the  buttons 
shown  in  Figure  4-1.  A set  of  two  threats  is  presented  at  the  top  of 
the  screen  and  moves  downward.  If  "full"  information  is  selected,  the 
differentiated  symbols  in  their  proper  location  move  down  the  screen. 

If  "outline"  information  is  selected,  the  missile  and  bird  symbols  are 
differentiated,  but  helicopters  and  airplanes  are  represented  by  a 
single,  non-differentiating  symbol  (AH).  Similarly,  "biological" 
information  will  use  a single,  non-differentiated  symbol  to  represent 
either  missile,  airplane,  or  helicopter.  "Location"  information  provides 
no  discrimination.  A symbol  denotes  the  location  but  not  the  identity 
of  threats.  "Left/right"  information  finally,  discriminates  the  obstacles 
using  the  standard  symbols,  but  locates  them  only  as  lying  in  the  left 
of  right  half  of  the  screen. 

The  task  moves  on  continuously,  just  as  an  airborne  mission  does. 
If  the  operator  does  not  select  an  information  choice  in  the  time 
allocated  (about  5 seconds),  location  information  is  provided  by  default. 
Following  information  receipt,  the  operator  must  make  an  action  selection 
before  the  threats  reach  the  decision  limit  (a  line  approximately  2/3  of 
the  distance  down  the  screen). 

4.3.3  Situational  Conditions.  The  stages  of  an  aircraft  mission  can  be 
characterized  by  such  factors  as  danger,  difficulty,  system  reliability, 
and  communications  security.  Accordingly,  the  task  simulation  was 
designed  to  include  many  of  the  same  factors.  The  situational  conditions 
are  not  considered  to  be  experimental  variables,  but  are  factors 
contributing  to  task  complexity.  The  conditions  are: 
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Degree  of  danger  --  This  is  the  distribution  of  possible 
threats  in  a given  phase.  A vector  of  prior  probabilities 
of  occurrence  of  the  4 threat  types  is  assigned  to  each 
phase. 

Costs  --  A different  cost  is  assigned  to  each  information 
choice.  This  is  the  number  of  points  expended  for  use  of 
the  information. 

Detection  --  The  increased  danger  on  the  succeeding  decision 
due  to  use  of  a given  information  source.  An  additional 
probability  of  loss  is  associated  with  each  of  the  threats. 

Delay  --  The  delay  in  seconds  before  actual  display  of  the 
information.  This  may  also  be  quantified  as  the  percent  of 
the  distance  to  the  action  limit  before  the  information  is 
displayed. 

Information  Accuracy  --  The  percentage  of  information 
transmissions  having  inaccuracies.  Inaccuracies  in 
location  will  be  present  on  a specified  percent  of  the 
decisions . 

Payoffs  --  Different  payoffs  in  points  are  made  for 
avoidance  of  or  damage  sustained  from  the  threats,  and 
for  successful  or  unsuccessful  aggressive  actions 
toward  the  threats.  Each  of  the  payoffs  vary  phase-by 
phase. 


The  presentation  of  conditions  is  organized  into  three  distinct 
mission  phases  — cruise,  surveillance,  and  aggression.  Each  phase  has 
set  levels  of  danger,  detection,  payoffs  and  information  accuracy.  For 
variability,  the  costs  and  delays  are  varied  smoothly  and  periodically 
within  each  phase.  All  conditions  are  displayed  to  the  subjects.  The 
mission  phases  and  their  associated  conditions  are: 

(1)  Cruise:  Low  danger  situation  with  high  accuracy  information 
available. 

(2)  Surveillance:  High  danger  situation  favoring  avoidance  actions; 
high  cost,  low  accuracy  information  is  present. 

(3)  Aggression:  High  danger  situation  oriented  toward  aggressive 
actions;  medium  accuracy  information  is  present. 

4.4  Decision  Model 

4.4.1  General . The  decision  faced  by  the  operator  is  a two-stage 
information/action  sequence,  as  shown  in  Figure  4-3.  The  decision  space 
is  fairly  large,  resulting  from  the  five  possible  information  choices, 

22  subsequent  action  choices  (avoid  or  attack  along  each  of  the  11 
pathways),  and  330  possible  states  (combination  of  threats).  A variety 
of  multidimensional  consequences  result  from  the  resulting  space,  stemming 
from  the  various  combinations  of  outcomes,  costs,  payoffs,  delays,  and 
future  impacts. 

A purely  analytical  formulation  of  this  is  intractable,  just  as 
it  is  for  most  operational  information  seeking  decisions.  Categorization 
is  an  obvious  means  of  reducing  the  complexity  of  the  decision.  Here  those 
elements  in  the  decision  similar  in  consequence  can  be  classified  together. 
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For  example,  the  number  of  states  can  be  reduced  from  330  categories  to 
6 categories  by  classifying  according  to  threat  combination  (ignoring 
specific  location).  The  probability  of  each  message  and  state  then  can 
be  calculated  from  the  prior  probabilities  of  each  of  the  threats  and  the 
information  source  characteristics.  The  actions  can  be  similarly 
categorized  as  avoid  or  attack  without  regard  to  location.  Probabilities 
of  each  outcome  type  --  avoidance,  damage,  missed  attack,  and  hit  — can 
be  established  by  observed  frequency.  To  do  so,  a probability  estimate 
must  be  associated  with  each  combination  of  information,  message,  action, 
and  state.  After  categorization,  90  such  combinations  are  present.  These 
probabi 1 i ties  were  determined  from  a series  of  pilot  system  tests,  and  were 
intended  to  be  representative  of  the  performance  of  the  typical  subject. 
Estimates  specific  to  each  subject  were  not  made.  The  consequence  levels 
(the  attribute  level  vector)  associated  with  a given  information  choice  in 
a given  situation  are  calculated  by  folding  back  the  decision  tree.  The 
favored  action  choice  after  receipt  of  a given  message  is  determined  in 
the  same  fashion. 


4.4.2  Decision  Attributes.  Five  consequence- related  attributes  are 
employed  in  the  decision  model.  The  attributes  are  the  following: 

(X-j)  Cost  - The  cost  of  the  communication  in  points  (costs  ranged 
from  0 to  15  points) . 


(X2)  Delay  - The  time  in  seconds  before  display  of  the  information 
(delays  ranged  from  0 to  4 seconds). 

(Xj)  Detection  - Increase  in  the  probability  of  damage  on  the 
subsequent  decision  ( the  probability  increase  ranged  from 
2 to  10  percent). 
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(X4)  Vehicle  loss  - Expected  (probability  weighted)  level  of  damage 
to  own  vehicle  (range:  2 to  14  points). 

(X5)  Offensive  gain  - Expected  level  of  damage  inflicted  on 
adversary  (range:  4 to  10  points). 

X4>  the  vehicle  loss  attribute  is  computed  according  to  the  following 
expression: 


x4  - l l 

message  action 
y a 


l {P(avoid  | y,a,z)  • Payoff  (avoid) 
state 
z 

-P(damage)  | y,a,z)  • Payoff  (damage)} 


X5>  the  offensive  gain  attribute,  is  calculated  in  an  identical 
manner  using  the  probabilities  and  payoffs  for  hit  and  miss. 


4.4.3  Weight  Estimation.  Two  forms  of  weight  estimation  were  employed 
in  the  study  --  adaptive  estimation  from  observed  behavior  and  off-line 
estimation  from  direct  elicitation.  Adaptive  estimation  was  performed 
using  the  pattern  recognition  method  described  in  Secion  3.3.4.  Prior  to 
the  experimental  session  each  subject  experienced  a training  session  of 
three  sequences  of  the  three  phases  (cruise,  surveillance,  and  aggression), 
A separate  5-element  weight  vector  was  maintained  for  each  of  the  three 
phases.  These  three  vectors  were  then  "frozen"  for  use  in  the  automated 
information  management  sessions  experienced  later. 

Off-line  estimation  followed  the  training  session  and  took  the 
form  described  in  Section  3.3.4.  Trial  experiences  with  the  direct 
elicitation  procedure  demonstrated  that  the  subjects  were  not  able  to 
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produce  different  weight  vectors  for  each  phase,  but  rather  followed  a 
single  overall  policy.  A single  5-element  vector  was  thus  elicitated 
from  each  subject  regarding  the  entire  training  session.  This  vector 
used  in  each  of  the  three  phases  for  automated  management  of  information. 

4.4.4  Evaluation.  The  evaluation  of  each  of  the  5 information  choices 
is  made  according  to  the  following  equation: 

5 

MAU  [I$]  = l K.  • Xi$ 
i=l 

where  MAU  [I$]  is  the  aggregate  (multi-attribute)  utility  of 
information  choice  Is,  Xis  is  the  level  of  attribute  i associated  with 
information  choice  I$,  (calculated  using  Equation  2-3)  and  is  the 
importance  weight  of  attribute  i.  It  should  be  noted  that  the  program 
did  not  have  access  to  the  true  state  of  the  environment. 

4. 5 Experimental  Procedure 

4.5.1  Experimental  Variables.  The  following  experimental  variables  and 
levels  were  tested: 

(1)  Model  Form  - Two  levels. 

(a)  Adaptive  estimation  of  weights  --  use  of  model -inferred 
attribute  weights  for  prediction,  evaluation,  and 
management  of  communications  decisions. 

(b)  Off-line  estimation  --  use  of  direct  elicitation  techniques 
to  define  attribute  weights  prior  to  the  task. 
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(2)  Communications  Management  - Two  levels. 

(a)  Unaided  operation  --  the  operator  makes  the  information 
and  control  choices  without  benefit  of  aiding. 

(b)  Automated  management  of  communications  --  model  based 
management  of  communications  (not  subject  to  operator 
override)  along  with  recommendation  of  avoidance/attack 
option. 

(3)  Speed  Stress  - Two  levels. 

(a)  Slow  speed  --  The  operator  has  a total  of  6 seconds  to 
observe  the  conditions  and  select  an  information  choice. 

(b)  Fast  speed  --  The  operator  has  4 seconds  to  observe  the 
conditions  and  select  an  information  choice. 

The  low  and  high  speed  stress  levels  were  chosen  empirically  to 
represent  two  extremes  of  load.  The  low  speed  rate  was  found  to  provide 
just  sufficient  time  for  consideration  of  all  factors.  The  high  speed  rate 
was  designed  to  rush  the  information  selection  somewhat,  but  not  to 
debilitate  the  ensuing  action  decision. 

4.5.2  Performance  Measures.  The  close  coupling  of  operator  and  aiding 
system  requires  evaluations  of  (1)  the  overall  system  performance  and 
(2)  the  performance  of  the  decision  model. 

System  Performance.  The  overall  system  performance  is  described 
using  a single  index,  the  score.  The  score  is  derived  from  the  number 
and  the  cost  of  errors  committed  and  the  communications  costs  expended: 


4-14 


SCORE  = (PAYOFFS)  - (PENALTIES  + COMMUNICATION  COSTS} 


The  score  is  presented  to  the  subject  as  a single  index  of 
performance,  and  his  compensation  depends  to  a large  extent  on  the  measure. 
The  complexities  of  having  speed  as  a second,  independent  measure  a^e 
avoided  by  presenting  the  task  at  a set  pace. 

Decision  Model  Performance.  The  effectiveness  of  the  decision  model 
is  evaluated  in  terms  of  behavioral  prediction,  operator  acceptance, 
construct  validity,  and  information  management  performance.  Prediction 
refers  to  the  ability  of  the  model  to  predict  operator  behavior  in  both 
the  information  and  action  decisions.  Outputs  of  the  adaptive  and  off-line 
estimated  models  were  compared  to  actual  operator  choices  during  the  unaided 
sessions.  Validity  tests  are  made  by  comparing  model  parameters  estimated 
by  the  adaptive  and  off-line  techniques. 

Multivariate  analysis  of  variance  was  used  to  analyze  the 
distribution  of  information  choices  and  the  vectors  of  importance  weights 
K. . The  dependent  variables  in  these  cases  are  (1)  the  frequency  with 
which  each  information  type  is  chosen,  and  (2)  the  vectors  of  by  subject 
and  phase. 

4.5.3  Subjects  and  Procedure.  The  twelve  subjects  participating  in  the 
study  were  recruited  from  nearby  universities  and  military  reserve  units. 
They  represented  the  type  of  personnel  who  might  interface  with  computer- 
aided  information  systems.  The  subjects  ages  ranged  from  21  to  39.  All 
had  two  or  more  years  of  college  experience.  Six  were  male  and  six  were 
female.  Six  subjects  had  experience  with  computer  systems  and  three  had 
flying  experience.  The  twelve  subjects  were  assigned  randomly  to  the  six 
groups. 


Each  subject  underwent  two  hours  of  orientation  and  practice  prior 
to  the  experimental  sessions.  The  practice  concluded  with  a slow  speed 
manual  session,  during  which  both  adaptive  estimation  and  off-line 
elicitation  of  attribute  weights  were  performed.  The  experimental  sessions 
consisted  of  three  complete  sequences  of  the  cruise,  surveillance,  and 
aggression  phases,  90  decisions  in  all.  A session  lasted  approximately 
35  minutes.  Each  subject  experienced  all  six  combinations  of  conditions 
in  a repeated  measures  design  (Figure  4-4).  The  subjects  were  paid  $5.00 
per  hour  and  were  given  a bonus  of  up  to  $5.00  per  hour  contingent  on 
performance. 


5.  EXPERIMENTAL  RESULTS  AND  DISCUSSION 


5.1  General  Observations 


The  choices  of  information  acquisition  and  action  selection  in  the 
simulation  were  found  to  be  sufficiently  varied  and  difficult  to  provide 
a good  test  of  model-based  information  management.  A wide  variety  of 
behaviors  were  observed  and  modeled.  The  task  simulation  was  also 
sufficiently  demanding  to  maintain  a high  level  of  subject  interest.  The 
subjects  learned  the  task  procedured  readily  and  by  the  end  of  the  training 
session,  could  effectively  handle  the  task  requirements  in  both  slow  and 
high  speed  conditions. 

5.2  Task  Performance 


Figure  5-1  shows  the  performance  score  ( the  payoffs  less  damages  and 
costs)  attained  under  each  of  the  three  forms  of  information  management  - 
manual,  automated/direct  elicitation,  and  automated/adaptive  estimation. 

The  Figure  also  shows  the  performance  scores  under  the  two  levels  of  speed 
stress. 


The  upper  portion  of  Figure  5-1,  dealing  with  the  low  speed 
conditions,  shows  a minor  (27%)  advantage  for  automated  management  with 
adaptive  estimation  (Auto  A)  over  the  other  two  forms  of  information 
management  (manual  and  Auto  D).  The  lower  portion  of  the  figure  shows 
a significantly  increased  aiding  in  the  higher  speed  condition.  Auto  A 
then  shows  a 60%  increase  in  performance  over  manual  selection,  and  Auto  D 
shows  a 29%  improvement  over  manual  (all  differences  significant  at  P<.05, 
Duncan  Multiple  Range  Test).  Much  of  the  improvement  in  aiding  appears  to 
be  due  to  the  significant  decrease  in  manual  performance  as  the  speed  stress 
increases.  The  automated  selection  performance  appears  to  remain  constant 
under  the  two  levels  of  speed  stress. 
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FIGURE  5-1:  THE  RELATIONSHIP  BETWEEN  INFORMATION 
SELECTION  MODE  AND  PERFORMANCE  SCORE  FOR: 

(A)  LOW  SPEED  STRESS,  AND  (B)  HIGH  SPEED  STRESS 
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Score  Components.  The  improvements  in  performance  with  aiding  were 
found  to  be  traceable  to  differences  in  both  the  costs  expended  and  the 
payoffs  attained.  Subjects  in  the  automated  selection  conditions  (Auto  A 
and  Auto  D)  incurred  significantly  greater  information  acquisition  costs 
than  subjects  using  manual  selection  (F(2,12)=10.73,  P< . 01 ) . Automated 
selection  also  resulted  in  significantly  greater  payoffs  (F(2,12)=13.2, 

P< . 01 ) compared  to  manual.  The  increase  in  payoffs  was  roughly  double  the 
increase  in  costs  expanded,  resulting  in  the  net  performance  increase 
observed  in  the  aided  conditions. 

The  quality  of  the  Auto  A performance  was  found  to  depend  on  the 
performance  on  the  training  session.  This  is  not  surprising,  since  the 
Auto  A selection  policy  is  based  on  the  training  session  behavior.  A 
correlation  of  .62  (P<.05)  was  observed  between  training  and  Auto  A scores. 

The  correlation  between  training  and  Auto  D scores  did  not  reach  significance. 

Subjective  Responses.  The  brief  questionnaires  administered  after 
each  session  provided  some  support  for  the  experimental  findings.  For 
example,  ten  of  the  twelve  subjects  expressed  a preference  for  manual 
information  selection  in  the  low  speed  stress  conditions.  In  the  high 
speed  stress  conditions,  nine  of  the  twelve  preferred  automated  selection 
(P<.05,  McNemar  test  for  significance  of  changes).  The  subject's  ratings 
of  their  performance  in  the  high  speed  stress  conditions  also  followed 
the  findings.  Performance  under  the  aided  conditions  was  seen  as 
significantly  higher  than  under  the  manual  conditions  (P<.01,  Wilcoxin 
matched-pairs  signed-ranks  test).  None  of  the  responses  regarding  comfort 
or  perceived  task  difficulty  reached  significance. 

5.3  Information  Seeking  Behavior 

It  should  be  noted  that  while  certain  trends  will  be  apparent  from 
the  analysis  of  sources  used,  it  is  net  possible  to  determine  the  optimal 
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policy  of  source  utilization  for  each  subject.  Each  operator  has  different 
control  capabilities  using  the  various  sources  and  different  responses  to 
the  time  delays  encountered.  Global  rather  than  individualized  analysis  of 
behavior  and  performance  will  be  attempted  here. 

Each  of  three  information  selection  modes  - manual.  Auto  A and 
Auto  D - exhibited  a different  distribution  of  choices  among  the  five 
information  options.  Figure  5-2  shows  in  histogram  from  these  choice 
distributions.  The  upper  histogram  shows  the  almost  uniform  use  of  the 
five  information  sources  with  manual  selection.  This  uniform  usage  may 
have  served  to  overemphasize  some  sub-optimal  sources.  The  Auto  A 
selection,  shown  in  the  middle  histogram,  exhibited  a more  peaked 
distribution  of  choices,  concentrating  on  the  biological,  location,  and 
full  information  choices  (see  section  4.3  for  a description  of  these 
choices).  The  Auto  D selection,  shown  in  the  lower  histogram,  was 
intermediate  in  diversity  among  the  sources. 

Within  the  Auto  A selection,  some  further  analysis  is  possible. 

The  average  policies  of  the  three  highest  scoring  subjects  are  compared 
with  the  average  policies  of  the  three  lowest  scoring  subjects  in  Figure 
5-3.  While  the  profiles  of  the  two  groups  are  quite  similar,  the  high 
scoring  group  (the  triangles  in  the  Figure)  appeared  to  place  a greater 
emphasis  on  cost  (A-j)  and  aggressive  gain  (A5)  than  did  the  low  scoring 
group. 

At  the  phase-by-phase  level,  a MANOVA  performed  on  the  adaptively 
estimated  attribute  vectors  revealed  significant  differences  in  policy 
between  the  cruise  and  surveillance  phases  (multivariate  F(5,7)=6.9,  P<.02) 
and  between  the  surveillance  and  aggression  phases  (multivariate  F(5,7)= 
16.4,  P<.01).  Between  cruise  and  surveillance  phases,  univariate  tests 
show  the  delay  weight  ( k2 ) to  decrease  (F(l ,11 )=28.3,  P< . 01 ) and  the 


5-  4 


' 


J .--Mr  ^ 


M IMMIAli 


ATTRIBUTE 


FIGURE  5-3:  AVERAGED  ATTRIBUTE  PROFILES 
FOR  LOW  SCORING  AND  HIGH  SCORING  GROUPS 


detection  weight  (k^)  to  increase  (F(l ,11 )=9.0,  P<.02).  Between 
surveillance  and  aggression  phases,  univariate  tests  showed  the  delay 
weight  (k^)  to  decrease  (F(l ,11)=10.7,  P<.01),  and  the  aggression  weight 
(k^)  to  decrease  (F(l ,11=21.9,  P<.01).  No  significant  changes  in  the  cost 
weight  ( k-j ) were  observed. 

The  presence  of  distinct  policies  for  each  phase  may  explain  much 
of  the  performance  advantage  of  the  Auto  A selection  over  the  Auto  D 
selection.  As  noted  earlier,  the  off-line  procedure  for  direct  elicitation 
resulted  in  only  a single  overall  vector  for  each  subject  instead  of  the 
phase-specific  vectors  obtained  by  the  adaptive  technique. 

5.4  Information  Value  Analysis 

It  was  brought  out  in  section  3.4  that  the  value  of  an  information 
source  could  be  calculated  from  the  operator's  policy  weights  and  the 
attribute  levels  encountered.  The  information  sources  in  the  task 
simulation  have  overlapping  function,  indicating  use  of  the  marginal 
contribution  (equation  3-6}  for  evaluation.  The  marginal  contribution  is 
the  incremental  value  of  a source  over  the  next  most  valued  source.  A 
source  thus  makes  a contribution  only  when  it  is  the  highest  valued  of  the 
set.  As  a corollary,  an  information  source  provides  zero  value  if  it  is 
never  chosen.  The  marginal  contributions  are  summed  over  all  decisions  in 
the  mission  to  arrive  at  an  overall  value  of  the  information  source. 

A test  of  the  usefulness  of  this  measure  is  the  correlation  between 
the  aggregate  information  value  of  each  source  and  the  score  actually 
obtained  using  that  source.  The  correlation  across  each  of  the  three 
phases  (15  scores  in  all)  was  .94  ( P< . 01 ) . The  correlation  did  not  reach 
significance  for  the  Auto  D estimation. 

I 
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By  way  of  comparison,  the  frequency  of  use  of  each  source  in  the 
manual  mode  correlated  .61  (P<.05)  with  the  scores  obtained  by  that  source. 
The  correlation  improved  to  .71  (P<.05)  when  the  marginal  information 
values  were  correlated  with  the  manual  scores  by  source.  This  indicates 
that  the  derived  information  value  estimate  may  be  more  useful  as  a design 
and  evaluation  tool  than  the  observed  frequency  of  use  of  an  information 


source. 


6.  CONCLUSIONS  AND  RECOMMENDATIONS 


6.1  Automated  Information  Management 

The  experimental  study  demonstrates  some  of  the  potential  of  on-line, 
adaptive  techniques  for:  (1)  automating  information  selection  and 
(2)  evaluating  alternative  information  configurations  in  advanced  aircraft 
operations.  In  the  advanced  aircraft  situation,  the  pilot  is  repeatedly 
required  to  make  complex,  subjective  decisions  regarding  information 
options.  Multi-attribute  modesl  using  either  pattern  recognition  techniques 
or  direct  elicitation  methods  for  estimation  were  seen  to  be  useful  for 
capturing,  analyzing,  and  automating  the  operator's  information  seeking 
policy. 


The  degree  of  aiding  provided  by  the  automated  information  selection 
was  found  to  depend  on  the  quality  of  the  training  behavior  and  on  the 
degree  of  speed  stress  in  the  task.  Not  surprisingly,  automated  information 
management  was  most  effective  when  it  was  based  on  parameters  estimated  from 
high  performance  manual  runs.  Also,  the  improvement  over  manual  information 
selection  was  greatest  in  situations  of  high  speed  stress.  In  support  of 
this,  subjects  expressed  a much  greater  perference  for  automated  information 
management  in  the  high  speed  stress  situations. 

Overall,  automated  information  management  using  the  off-line  direct 
elicitation  technique  was  not  found  to  be  quite  as  effective  as  the  adaptive 
technique.  This  may  have  been  due  to  the  ability  to  elicit  only  a single 
vector  of  weights  for  all  mission  phases.  It  is  possible  that  other  methods 
of  direct  elicitation  - paired  comparisons,  indifference  curves,  probability 
wagers,  etc.  - may  be  sensitive  enough  to  differentiate  between  sub- task 
policies,  albeit  at  longer  elicitation  time  and  difficulty.  The  elicitation 
technique  used  was  not  offered  as  the  optimal  method,  but  merely  illustrative 
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of  the  procedures  available.  It  is  anticipated  that  in  those  situations 
where  few  decisions  are  available  for  training,  or  the  decisions  are  too 
complex  for  effective  choice,  the  off-line  procedures  would  be  preferred. 

Increased  aiding  with  the  adaptive  technique  is  also  possible. 

The  probability  estimation  programs  and  attribute  weights  were  both 
"frozen"  during  the  experimental  sessions.  Dynamic  estimation  of  outcome 
probabilities  specific  to  each  subject  should  improve  the  accuracy  of  the 
attribute  level  estimates  and,  in  turn,  result  in  more  effective  information 
selection.  Similarly,  allowing  the  attribute  weights  to  adjust  dynamically 
with  the  task  demands  should  improve  performance. 

6.2  Information  Value  Estimation 


The  mul ti-attribute  model  was  found  to  provide  a useful  framework 
for  ascertaining  the  value  or  contribution  of  each  information  source. 

The  marginal  contribution  estimated  for  each  information  source  in  the 
experimental  study  proved  to  be  a good  estimator  of  the  actual  score 
attributable  to  that  source.  This  information  value  was  specific  to 
the  individual  decision  maker  and  the  sequence  of  task  circumstances 
encountered. 

The  Adaptive  model  employed  in  this  study  was  constrained  to  an 
application  of  moderate  complexity  - multiple  criteria,  probabilistic 
consequences,  and  time-varying  behavior.  Extension  of  the  domain  of 
application  to  more  complex  circumstances  - limited  resources,  continued 
sampling  of  information,  and  multiple  tasks  - is  scheduled  for  the  coming 
year.  Accomplishment  of  these  extensions  should  allow  application  to  a 
variety  of  operational  systems. 
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